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LINE BASED IMAGE MATCHING METHOD 



BACKGROUND OF THE INVENTION 



1. 



Field of the Invention 



The present invention relates to an image matching method, and more 
particularly, to a line based image matching method for retrieving a model 
5 image indexed by similar shape describers to a query image from an image 
database indexed by line based shape describers. The present application is 
based on Korean Patent Application No. 2000-82756, filed December 27, 



2000, which is incorporated herein by reference. 

A shape descriptor, as a basic image pattern recognition describer, is 

10 based on an automatically extractable lower abstraction level description. 
Recent approaches have been made on algorithms for describing the shape of a 
particular object from an image, and measuring a degree of matching accuracy 
or similarity based on the shape descriptors. These algorithms are sufficient to 
describe the shape of a particular object but not to recognize the shape of an 

15 arbitrary object. The shape descriptors suggested by the MPEG-7 standard are 
obtained with feature points found through a variety of modifications of a 
given object. There are many kinds of shape descriptors. The experiment 
Model 1 (XM) of the MPEG-7 standard suggest two types of shape 
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descriptors: a Zernike moment shape descriptor and a Curvature scale space 
shape descriptor. As for the Zernike moment shape descriptor, Zernike base 
functions are defined for a variety of shapes of an object to investigate the 
distribution of the shapes within an image. Then, the image in a 
5 predetermined size is projected over all the base functions, and the resultant 
values are used as the shape describers. As for the curvature scale space shape 
descriptor, the contour of a model image is extracted, and changes of curvature 
points of an image along the contour are expressed as peak values on a scaled 
space. Then, the location of the peak values are expressed as a 2-dimensional 
10 vector. The extraction of the former shape descriptor can be applied only 
when each shape of the image and the base function have a similar scale. 
Also, the size of input image is restricted. On the other hand, a drawback of 
the latter shape descriptor extraction lies in that the shapes are extracted from 
only a single object. 

15 According to a conventional shape descriptor based image matching 

method, accurate extraction of an object from a query image is needed prior to 
retrieving of a model image indexed by similar shape describers to the query 
image from a database. In other words, if the object cannot be accurately 
extracted, retrieving of a model image for the conventional image matching is 

20 impossible. 

Therefore, there is a need for a simple image matching method capable 
of retrieving a model image indexed by similar shape describers to a query 
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image from an image database indexed by line based shape descriptors, 
without need for accurate object extraction. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide a line based image 
5 matching method capable of retrieving a model image indexed by similar 
shape descriptors to a query image from a model image database indexed by 
line based shape describers, without need for accurate object extraction. 

The object of the present invention is achieved by a line based image 
matching method comprising: collecting line information of a query image and 
10 model images; defining the binary relation between the lines of the query 
image and the lines of the model images; measuring the compatibility 
coefficients of the node-label pairs of the query and model images based on 
the binary relation; and measuring the similarity between the query and model 
images on the basis of continuous relaxation labeling using the compatibility 
15 coefficient. 

It is preferable that the line information of each of the query and model 
images is expressed by shape descriptors. 

It is preferable that the model images are retrieved from a database 
indexed by shape descriptors determined by: extracting the skeleton of a 
20 model image; thinning the skeleton; concatenating corresponding pixels based 
on the extracted skeleton to obtain a set of lines; and normalizing the set of 
lines to determine the normalized set of lines as the shape descriptors. 
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It is preferable that the binary relation is invariant with respect to 
rotations, scale changes and translations. In this case, the binary relation may 
include at least one of an angular difference between two lines, a ratio of the 
lengths, a relative location, and a relative distance. 
5 It is preferable that measuring the compatibility coefficients of the 

node-label pairs based on the binary relation comprises: measuring the binary 
relation, denoted by £,y, for two nodes / and j within the set of lines of the 
query image; measuring the binary relation, denoted by S^v? for two labels X 
and X' within the set of lines for each of the model images; and measuring the 

10 compatibility coefficients, denoted by n/X 9 A,'), for the node-label pairs of the 
query and each of the model images. The compatibility coefficients r^X, A,') 
as a measure of the strength of compatibility between the node-label pairs may 
have high values corresponding to compatibility and low values corresponding 
to incompatibility. The compatibility coefficients r,y(A,, X') may be determined 

15 as 1 if the binary relation of a node pair (/, j) of the query image coincides with 
the binary relation of a label pair (X, A,'). The compatibility coefficients ry{X 9 
X 7 ) can be expressed as: 

20 

where pVJ.W) = (£ l^'f&'ir)"" . "K" denotes the number of elements 
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of a character vector for a defined binary relation; and "p" is a measure of the 
difference in compatibility on between node-label pairs. 

It is preferable that, before measuring the similarity on the basis of the 
continuous relaxation labeling, the line based image matching method further 
5 comprises assigning a uniform initial probability to a predetermined number of 
upper node-label pairs in which the sums of the highest degree of support by 
each adjacent label for the nodes are within the range of an upper level, the 
initial probability being close to the final probability. After assigning the 
uniform initial probability to the upper node-label pairs, the line based image 
1 0 matching method may further comprise defining a probability update element 
for the continuous relaxation labeling as: 

j *. 

15 where p {k) (/i') denotes the node-to-label correspondence probability, and k 
denotes the number of iterations needed. After defining the probability update 
element, the line based image matching method may further comprise 
updating the probability on the basis of the Zucker's theory using 



where = £ *I r£»{W)p)*W . and = £ p\ k \A)q\ k \X). 
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It is preferable that measuring the similarity between the query and 
model images comprises: calculating the sum of the distances between 
corresponding node-to-label pairs of the sets of lines for the query and each of 
the model images; and determining the reciprocal of the sum of the distances 
5 as the similarity between corresponding two images. The distances may be 
measured using the Euclidean distance or the Housdorff distance. 



The above object and advantages of the present invention will become 
more apparent by describing in detail preferred embodiments thereof with 
1 0 reference to the attached drawings in which: 

FIG. 1 is a flowchart illustrating the main steps of a line based image 
matching method according to a preferred embodiment of the present 
invention; 



1 5 the binary relation between two lines; 

FIG. 2B illustrates an angle between the line segment connecting the 
middle points of two lines, and one of the two lines, as another example of the 
binary relation between two lines; 



BRIEF DESCRIPTION OF THE DRAWINGS 



FIG. 2A illustrates a difference in slope of two lines as an example of 



FIG. 3A shows an example of a query image; 



20 



FIG. 3B illustrates the query image of FIG. 3A after a line 



approximation of the present invention; 
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FIG. 3C illustrates some lines selected from the query image of FIG. 
3B by a user; 

FIG. 4A illustrates an example of a model image corresponding to the 
query image which was retrieved in a simulation test; 
5 FIG. 4B illustrates the model image of FIG. 4A after a line 

approximation process; 

FIG. 5 A illustrates another example of a model image; and 

FIG. 5B illustrates the model image of FIG. 5A after a line 
approximation process. 
10 DETAILED DESCRIPTION OF THE INVENTION 

A preferred embodiment of a line based image matching method 
according to the present invention is illustrated in FIG. 1 . Referring to FIG. 1 , 
line information on a query image and model images are collected (step 102). 
Here, it is assumed that the model images are stored in a database, and indexed 
15 by line based shape descriptors. These line based shape descriptors can be 
extracted by the shape descriptor extraction method which has been filed on 
October 21, 2000, by the present applicant and assigned Korean Patent 
Application No. 00-62163, and is incorporated herein by reference. Shape 
describers can be extracted from a query image by the shape descriptor 
20 extraction method. The shape descriptors extracted by the shape descriptor 
extraction method includes a set of lines. Line information on the query image 
and model images can be collected from the set of lines. 
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Binary relations between the lines of the query image and model 
images are defined (step 104). The intrinsic attributes of a single line, such as 
slope, length or location of the middle point, and binary relations between two 
lines, such as angular difference, ratio of the lengths, relative location, or 
relative distance between two lines, can be used as the features of the lines. In 
the present embodiment, the binary relations that are invariant to rotations, 
scale changes, and translations are used as the lines features. FIG. 2A 
illustrates a difference in slope between two lines as an example of the binary 
relation. As shown in FIG. 2A, the relative position of the two lines can be 
expressed by the difference (9) in the slopes of the two lines. FIG. 2B 
illustrates an angle between the line segment connecting the middle points of 
the two lines, and one of the two lines, as another example of the binary 
relation between two lines. As shown in FIG. 2B, the relative location 
between two lines can be expressed by the angle (<|>). Although not illustrated 
in the drawings, a ratio (r) of lengths between two lines can be used as a 
binary relation between the two lines. 

In the line based image matching method according to the present 
invention, the correspondence between a set of lines for the query image, and 
a set of lines for the model images, are determined based on continuous 
relaxation labeling. For the continuous relaxation labeling, the initial 
probability is updated on the basis of compatibility coefficients, and a more 
recent labeling status. Once the initial probability is assumed, the probability 
is updated depending on the compatibility coefficients. The compatibility 
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coefficients are used in modeling possible matching conditions to reach a 
satisfactory labeling. Accurate labeling close to a real query image is 
important. 

Next, in step 106, the compatibility coefficients between node-label 
5 pairs of the query and model images are determined based on the binary 
relations defined in step 104. The procedures of step 106 are as follows. First, 
a binary relation value ^ between two nodes v, and vj within the set of lines 
for the query image is calculated based on their binary relations. Here, i and j 
y3 denote the identifiers of the nodes of the query image, and are not identical to 

03 1 0 each other. A binary relation value £>xk> between two labels X and X } within the 
set of lines for each model image is calculated based on their binary relations. 
7 Here, X and A,' denote the identifiers of the labels of the model image. The 

«=:;=; 

Ep strength of compatibility between the node pairs of the query image and the 

p label pairs of the model image is expressed as the compatibility coefficient 

^ 15 n/X, A,'). The compatibility coefficient r,y(A,, X') is determined to be between 0 
and 1 based on the binary relations of the query image and model images. The 
compatibility coefficient ry(X, A,') measures the strength of compatibility 
between node-label pairs: high values correspond to compatibility and low 
values correspond to incompatibility. For example, in considering the 
20 compatibility of the node-label pair (v /3 X), if the compatibility of label X\ for 
the node v, is higher than that of label X2, the compatibility coefficient is 
defined to satisfy the following relation: 
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...(1) 



When the binary relation of a node pair (/, j) of the query image 
coincides with the binary relation of a label pair (A,, A,'), the coefficient r tJ {k 9 
A,') is determined as 1. In the present embodiment, the compatibility 
5 coefficient r 7 y(A,, A, 5 ) is expressed as 



elements of a character vector for a defined binary relation. In formula (2), 
"p" is a measure of the difference in compatibility between node-label pairs. 

Next, the similarity is determined based on the continuous relaxation 
labeling with the compatibility coefficients (step 108). As previously 

1 5 described, in a continuous relaxation labeling process, the initial probability is 
updated on the basis of compatibility coefficients, and a more recent labeling 
status. Once the initial probability is assumed, the probability is updated 
depending on the compatibility coefficients. The compatibility coefficients 
are used in modeling possible matching conditions to reach a satisfactory 

20 labeling. Accurate labeling close to a real query image is important. Step 
108, performed based on the continuous relaxation labeling using the 
compatibility coefficients, involves a probability updating step, a 
normalization step, and a similarity determination step. First, a uniform initial 




...(2) 

K 

where p(fJ,W) = (£ )"° ■ and "K" denotes the number of 
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probability is assigned to a predetermined number of upper node-label pairs in 
which the sums of the highest degree of support by each adjacent label for the 
nodes are within the range of an upper level, wherein the initial probability 
must be close to the final probability. 
5 A probability update element for the continuous relaxation labeling is 

defined as 

10 where Pj k) (JL') denotes the node-to-label correspondence probability, and k 
denotes the number of iterations needed. The probability is updated as follows 
on the basis of the Zucker's theory which ensures efficient convergence result: 

where = £ a£ /f>0M ')/><*> {V) , and ^ = £ p< ki Wqf l \X). 

j X X 

...(3) 

The range of each element of the character vector does not coincide 
15 with other elements, and thus there is a need for normalization. Information 
on lines to be matched with a query image can be clearly obtained by the 
node-to-label correspondence based on the relaxation labeling. One of the 
lines from the set for the query image is selected, and the remainder of the 
lines is normalized with respect to the selected line in terms of size and 
20 location. In particular, an arbitrary line is selected from the set of lines for a 
query image. The length of the selected line is assigned 1, and a spatial 
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transform function is defined such that the selected line is aligned on the X- 
axis of an orthogonal coordinates system. Then, all the lines for the query 
image is transformed by the spatial transform function. This spatial 
transformation is carried out for all the lines of the model images 
5 corresponding to the selected lines. 

Then, the similarity between the query image and model images are 
determined. In the present embodiment, the similarity is determined by 
calculating the sum of the node-to-label distances of the query and each of the 
model images. A variety of techniques can be applied to determine the 

10 similarity. For example, if the node-to-label correspondence between the 
query image and the model images is apparent, and if the normalization with 
respect to rotations and scale changes is carried out for the corresponding 
lines, the distances between the nodes and labels can be directly measured 
using the Euclidian distance. Alternatively, the distances can be measured 

1 5 using the Housdorff distance, which ensures a more accurate similarity result. 
In the present embodiment, the similarity between the query image and a 
model image is measured as follows. The distances between the 
corresponding nodes and labels are measured for the two line sets of the query 
image and each of the model images, using the Euclidian distance. Then, the 

20 sum of all the distances is calculated, and then the similarity between two 
images is determined with the reciprocal of the sum. A model image having 
the highest degree of similarity is retrieved (step 1 10). 
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A simulation experiment was carried out based on the above- 
mentioned line based image matching method. Edges of a query image were 
extracted, and the extracted edges were approximated into lines to express the 
query image with the lines. Then, some of the lines of the query image were 
5 selected by a user, and then matched with model images using information on 
the selected lines. An example of a query image is shown in FIG. 3A. FIG. 
3B illustrates the query image of FIG. 3A after the above-mentioned line 
approximation. FIG. 3C illustrates some lines selected from the query image 
of FIG. 3Bby user. 

1 0 FIG. 4A illustrates an example of a model image corresponding to the 

query image which was retrieved in the simulation test. FIG. 4B illustrates the 
model image of FIG. 4A after line approximation. As shown in FIG. 4B, the 
retrieved model image approximated with lines, as indicated by reference 
numeral 402, includes similar lines to the line approximated query image, as 

15 shown in FIG. 3C. 

FIG. 5A illustrates another example of a model image, and FIG. 5B 
illustrates the model image of FIG. 5 A after line approximation. As shown in 
FIG. 5B, the line approximated model image, as indicated by reference 
numeral 502, also includes similar lines to the line approximated query image, 

20 as shown in FIG. 3C. 

As previously mentioned, in the line based image matching method 
according to the present invention, a model image with similar shape 
descriptors to a query image can be retrieved from an image database indexed 
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by line based shape descriptors, without need for object extraction from the 
query image. 

The line based image matching method according to the present 
invention may be embodied as a program executable on a personal or server 
computer. A functional program, code and code segments, used to implement 
the present invention, can be derived by a skilled computer programmer from 
the description of the invention contained herein. The program can be stored 
in a computer usable medium, including but not limited to storage media such 
as magnetic storage media, optically readable media, and carrier waves. 
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